Image capture devices for autonomous mobile robots and related systems and methods

ABSTRACT

An autonomous cleaning robot includes a drive system to support the autonomous cleaning robot above a floor surface, an image capture device positioned on the autonomous cleaning robot to capture imagery of a portion of the floor surface forward of the autonomous cleaning robot, and a controller operably connected to the drive system and the image capture device. The drive system is operable to maneuver the autonomous cleaning robot about the floor surface. The controller is configured to execute instructions to perform operations including initiating, based on a user-selected sensitivity and the imagery captured by the image capture device, an avoidance behavior to avoid an obstacle on the portion of the floor surface.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of, and claims priorityto, U.S. patent application Ser. No. 16/588,295, filed on Sep. 30, 2019.The disclosure of the foregoing application is incorporated herein byreference in its entirety for all purposes.

TECHNICAL FIELD

This specification relates to image capture devices for autonomousmobile robots and related systems and methods.

BACKGROUND

Autonomous mobile robots include autonomous cleaning robots thatautonomously perform cleaning tasks within an environment, e.g., a home.Many kinds of cleaning robots are autonomous to some degree and indifferent ways. A cleaning robot can include a controller configured toautonomously navigate the robot about an environment such that the robotcan ingest debris as it moves. The cleaning robot can include a sensorfor avoiding obstacles in the environment.

SUMMARY

An autonomous cleaning robot can include a camera facing in a forwarddirection of the robot. The present disclosure describes various waysthat the forward facing camera can be used in operations of the robot.Based on imagery captured by the camera, the robot can behave in certainways in response to obstacles and features ahead of the robot. Forexample, the camera can be used to capture imagery of the floor surface,allowing the robot to detect area rugs on the floor surface. The robot,in response to detecting an area rug on a portion of the floor surface,can initiate a behavior to move along the floor surface in a manner thatreduces the risk of error as the robot moves over the portion of thefloor surface with the area rug. Furthermore, the imagery captured bythe camera can also be used to provide the user with information aboutan environment of the robot. For example, imagery captured by the cameracan be used to provide a map of the environment that indicates floortypes of different portions of the floor surface in the environment. Theimagery can also be used to provide information about features alongwalls of the environment, such as windows, paintings, and the like.Moreover, the map can provide information on locations of obstacles inthe environment, and the user can operate a user computing device toselect a sensitivity of the robot to detection of these obstacles in theenvironment. When the selected sensitivity is high, the robot tends toinitiate an obstacle avoidance behavior at a distance longer than thedistance when the selected sensitivity is low.

Advantages of the foregoing may include, but are not limited to, thosedescribed below and herein elsewhere.

Implementations described herein can improve the experience for users ininteracting with autonomous mobile robots. Imagery captured by a cameraof an autonomous mobile robot can provide information to a user about anenvironment of the robot such that the user can make informed decisionsfor controlling operations of the robot. The information can bepresented visually on a user computing device in the form of arepresentation of a map. For example, a representation of a mappresented to the user can indicate a floor surface type of a portion ofa floor surface in the environment. To visually represent the floorsurface type, a user computing device can present at least a portion ofthe imagery that represents the portion of the floor surface.Alternatively, the floor surface type can be identified from theimagery, and then the user computing device can present a map thatcontains a representation of the floor surface type. The visualrepresentations presented to the user through the user computing deviceallow the user to easily obtain information about the environment, andthen use this information to control operations of the robot.

In further examples, objects in the imagery captured by the camera ofthe robot can be identified such that representations of these objectscan be presented to the user. These representations can be presented tothe user on a user computing device to allow the user to track theobjects encountered by the robot during its operations in theenvironment. As a result, the user can easily respond to detection ofcertain objects in the environment, e.g., cords, clothing, or the like,by tidying up or cleaning up these objects so that the robot does notencounter them again in subsequent cleaning operations.

Implementations described herein can increase the amount of control thatthe user has over operations of an autonomous mobile robot. As the robotdetects objects in the environment, the robot can perform obstacleavoidance behavior to avoid the objects. In this behavior, the robot cantravel along the floor surface such that the robot remains a certaindistance from an object as the robot avoids the object. The user, inimplementations described herein, can select this distance or otherwiseselect a sensitivity of the robot to avoiding obstacles. Thus while therobot can autonomously perform operations in the environment, the userstill has control over certain behaviors of the robot, allowing the userto control operations in a way that is suitable for the unique featuresof the environment in which the robot is operating. The user can controlthe sensitivity so that the robot can cover a greater amount of area inthe environment without significantly increasing the rate that the robotexperiences error conditions.

Implementations described herein can also provide an intuitive way forthe user to control the operations of the robot. The user can interactwith a visual representation of the environment that intuitivelyprovides the user with information about the environment. Because thevisual representation can be constructed based on imagery captured ofthe environment, the visual representation can better correspond withthe actual visual appearance of the environment. In addition, the userinterface controls for adjusting, for example, the sensitivity of therobot to detection of objects in the environment, can be intuitivelyoperated by the user.

Implementations described herein can allow an autonomous cleaning robotto clean an area rug with a reduced risk of experiencing an errorcondition associated with the area rug. For example, an autonomouscleaning robot may ingest tassels of, a corner of, or other portions ofthe area rug when navigating over the area rug and then experience anerror condition. Using imagery captured by the camera of the autonomouscleaning robot, the robot can initiate a movement pattern relative tothe area rug that reduces the risk that the robot ingests a portion ofthe area rug.

Implementations described herein can provide autonomous mobile robotsthat appear more intelligent to human users as the robots travel aroundtheir environments. For example, as an autonomous mobile robot with afront-facing camera moves about its environment, the front-facing cameracan see a portion of the floor surface ahead of the robot such that therobot can initiate behaviors in anticipation of objects ahead of therobot. As a result, the robot can initiate behaviors well beforecontacting an object or being adjacent to the object, thus providingtime and physical space to respond to detection of the object by thecamera. The robot can, for example, slow down or turn relative to theobject, and thus provide the appearance that the robot is intelligentlyresponding to the object.

In one aspect, a mobile computing device includes a user input device,and a controller operably connected to the user input device. Thecontroller is configured to execute instructions to perform operationsincluding receiving, from the user input device, data indicative of auser-selected sensitivity for obstacle avoidance by an autonomouscleaning robot, and initiating transmission of the data indicative ofthe user-selected sensitivity to the autonomous cleaning robot such thatthe autonomous cleaning robot initiates an obstacle avoidance behaviorto avoid an obstacle on a portion of a floor surface based on imagerycaptured by an image capture device of the autonomous cleaning robot andthe user-selected sensitivity.

In another aspect, an autonomous cleaning robot includes a drive systemto support the autonomous cleaning robot above a floor surface, an imagecapture device positioned on the autonomous cleaning robot to captureimagery of a portion of the floor surface forward of the autonomouscleaning robot, and a controller operably connected to the drive systemand the image capture device. The drive system is operable to maneuverthe autonomous cleaning robot about the floor surface. The controller isconfigured to execute instructions to perform operations includinginitiating, based on a user-selected sensitivity and the imagerycaptured by the image capture device, an avoidance behavior to avoid anobstacle on the portion of the floor surface.

In a further aspect, a method includes capturing, by an image capturedevice on an autonomous cleaning robot, imagery of a portion of a floorsurface forward of the autonomous cleaning robot, the portion of thefloor surface including at least a portion of a rug, and maneuvering theautonomous cleaning robot onto the rug along a path selected based onthe imagery of the portion of the floor surface.

In a further aspect, an autonomous cleaning robot includes a drivesystem to support the autonomous cleaning robot above a floor surface,an image capture device positioned on the autonomous cleaning robot tocapture imagery of a portion of the floor surface forward of theautonomous cleaning robot, and a controller operably connected to thedrive system and the image capture device. The drive system is operableto maneuver the autonomous cleaning robot about the floor surface. Theportion of the floor surface includes at least a portion of a rug. Thecontroller is configured to execute instructions to perform operationsincluding maneuvering the autonomous cleaning robot onto the rug along apath selected based on the imagery of the portion of the floor surface.

In a further aspect, an autonomous cleaning robot includes a drivesystem to support the autonomous cleaning robot above a floor surface,an image capture device positioned on the autonomous cleaning robot tocapture imagery of a portion of the floor surface forward of theautonomous cleaning robot, and a controller operably connected to thedrive system and the image capture device. The drive system is operableto maneuver the autonomous cleaning robot about the floor surface. Thecontroller is configured to execute instructions to perform operationsincluding maneuvering the autonomous cleaning robot at a first speedalong a first portion of the floor surface toward a second portion ofthe floor surface, detecting the second portion of the floor surfacebased on the imagery captured by the image capture device, andmaneuvering the autonomous cleaning robot at a second speed along thefirst portion of the floor surface toward the second portion of thefloor surface after detecting the second portion of the floor surface.The second portion of the floor surface has a lower elevation than thefirst portion of the floor surface. The second speed is less than thefirst speed.

In a further aspect, an autonomous cleaning robot includes a drivesystem to support the autonomous cleaning robot above a floor surface,and a controller operably connected to the drive system. The drivesystem is operable to maneuver the autonomous cleaning robot about thefloor surface. The controller is configured to execute instructions toperform operations including maneuvering the autonomous cleaning robotat a first speed along a first portion of the floor surface toward asecond portion of the floor surface, and after the autonomous cleaningrobot is within a distance from the second portion of the floor surface,maneuvering the autonomous cleaning robot at a second speed along thefirst portion of the floor surface based on the autonomous mobile. Thesecond portion of the floor surface has a lower elevation than the firstportion of the floor surface. The second speed is less than the firstspeed.

In a further aspect, a method includes maneuvering an autonomouscleaning robot at a first speed along a first portion of a floor surfacetoward a second portion of the floor surface, detecting, using an imagecapture device positioned on the autonomous cleaning robot to captureimagery of a portion of the floor surface forward of the autonomouscleaning robot, the second portion of the floor surface, and maneuveringthe autonomous cleaning robot at a second speed along the first portionof the floor surface toward the second portion of the floor surfaceafter detecting the second portion of the floor surface. The secondportion of the floor surface has a lower elevation than the firstportion of the floor surface. The second speed is less than the firstspeed.

Implementations can include one or more features below or describedherein elsewhere. Implementations can include combinations of the belowfeatures.

In some implementations, the user-selected sensitivity can be indicativeof a distance threshold such that the autonomous cleaning robotinitiates the obstacle avoidance behavior based on a distance betweenthe obstacle and the autonomous cleaning robot being no more than thedistance threshold. In some implementations, receiving the user-selectedsensitivity can include receiving data indicative of a user selection ofthe distance threshold.

In some implementations, the user-selected sensitivity is indicative ofa likelihood threshold such that the autonomous cleaning robot initiatesthe obstacle avoidance behavior based on a likelihood of a presence ofthe obstacle on the portion of the floor surface being no less than thelikelihood threshold. In some implementations, the likelihood of thepresence of the obstacle can be determined based on the imagery capturedby the image capture device.

In some implementations, the mobile computing device can further includea display operably connected to the controller. The obstacle can berepresented in the imagery captured by the image capture device. Theoperations can include receiving, from the autonomous cleaning robot,data representative of the imagery, and presenting, on the display, arepresentation of the obstacle based on the data representative of theimagery.

In some implementations, the mobile computing device can include adisplay operably connected to the controller. The operations can includepresenting, on the display, representations of obstacles present inimagery by the image capture device of the autonomous cleaning robot,the representations of the obstacles including a representation of theobstacle.

In some implementations, the user-selected sensitivity can correspond toa user-selected distance threshold, and initiating the avoidancebehavior to avoid the obstacle can include initiating the avoidancebehavior based on a distance between the obstacle and the autonomouscleaning robot being no more than the distance threshold.

In some implementations, the user-selected sensitivity can correspond toa likelihood threshold, and initiating the avoidance behavior to avoidthe obstacle can include initiating the avoidance behavior based on alikelihood of a presence of the obstacle on the portion of the floorsurface being no less than the likelihood threshold.

In some implementations, the operations can include initiatingtransmission of data indicative of images captured by the image capturedevice to cause a remote user device to present representations ofobstacles present in the images.

In some implementations, the imagery of the portion of the floor surfacecan be indicative of a location of a tassel of the rug, and maneuveringthe autonomous cleaning robot onto the rug can include maneuvering theautonomous cleaning robot onto the rug along the path such that theautonomous cleaning robot avoids the tassel.

In some implementations, the path can be a first path. The imagery ofthe portion of the floor surface can be indicative of a direction alongwhich a tassel of the rug extends along the floor surface. Theoperations can further include maneuvering the autonomous cleaning robotoff of the rug along a second path such that the autonomous cleaningrobot moves over the tassel in a direction substantially parallel to thedirection along which the tassel extends.

In some implementations, the imagery of the portion of the floor surfacecan be indicative of a location of a corner of the rug. Maneuvering theautonomous cleaning robot onto the rug can include maneuvering theautonomous cleaning robot onto the rug along the path such that theautonomous cleaning robot avoids the corner of the rug.

In some implementations, the imagery can include images. Maneuvering theautonomous cleaning robot onto the rug along a path selected based onthe imagery of the portion of the floor surface can include maneuveringthe autonomous cleaning robot onto the rug along the path selected basedon a location of an edge of the rug represented in the plurality ofimages. In some implementations, maneuvering the autonomous cleaningrobot onto the rug along the path selected based on the location of theedge of the rug represented in the plurality of images can includemaneuvering the autonomous cleaning robot onto the rug along the pathselected based on a stitched image representation of the floor surfacegenerated from the plurality of images.

In some implementations, the autonomous cleaning robot can include arotatable member on a bottom portion of the autonomous cleaning robot,and a motor to rotate the rotatable member to direct debris into aninterior of the autonomous cleaning robot. The operations can furtherinclude operating the motor to rotate the rotatable member at a firstspeed of rotation as the autonomous cleaning robot moves about a portionof the floor surface off of the rug, and operating the motor to rotatethe rotatable member at a second speed of rotation as the cleaning robotmoves from the portion of the floor surface off of the rug to a portionof the floor surface on the rug. The second speed of rotation can beless than the first speed of rotation. In some implementations, thesecond speed of rotation is zero. In some implementations, theoperations can further include operating the motor to rotate therotatable member at third speed of rotation as the cleaning robot movesabout the rug, and operating the motor to rotate the rotatable member ata fourth speed of rotation as the cleaning robot moves from the portionof the floor surface on the rug to the portion of the floor surface onthe rug. The third speed of rotation can be greater than the secondspeed of rotation. The fourth speed of rotation can be greater than thesecond speed of rotation.

In some implementations, maneuvering the autonomous cleaning robot atthe second speed along the first portion of the floor surface afterdetecting the second portion of the floor surface can include initiatingreduction of a speed from the autonomous cleaning robot from the firstspeed to the second speed based on determining, from the imagerycaptured by the image capture device, the autonomous cleaning robot isno more than a distance from the second portion of the floor surface. Insome implementations, the distance can be between 50% to 300% of alength of the autonomous cleaning robot.

In some implementations, the imagery captured by the image capturedevice can represent at least a portion of the second portion of thefloor surface.

In some implementations, the autonomous cleaning robot can include asingle image capture device corresponding to the image capture device.

In some implementations, the image capture device can be directed at anangle between 10 and 30 degrees above the floor surface. In someimplementations, a horizontal field of view of the image capture devicecan be between 90 and 150 degrees.

In some implementations, the autonomous cleaning robot can include acliff sensor disposed on a bottom portion of the autonomous cleaningrobot. The cliff sensor can be configured to detect the second portionof the floor surface as the bottom portion of the autonomous cleaningrobot moves over the second portion of the floor surface. In someimplementations, the operations can include maneuvering the autonomouscleaning robot along the first portion of the floor surface away fromthe second portion of the floor surface as the cliff sensor detects thesecond portion of the floor surface.

The details of one or more implementations of the subject matterdescribed in this specification are set forth in the accompanyingdrawings and the description below. Other potential features, aspects,and advantages will become apparent from the description, the drawings,and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a perspective view of an autonomous cleaning robot in anenvironment.

FIG. 2 is a side schematic view of an autonomous cleaning robot in anenvironment.

FIG. 3A is a bottom view of an autonomous cleaning robot.

FIG. 3B is a top perspective view of the robot of FIG. 3A.

FIG. 4 is a side schematic view of an autonomous cleaning robot with animage capture device in an environment.

FIG. 5 is a diagram of a communications network.

FIG. 6 is a flowchart of a method of producing a map of an environment.

FIG. 7A is a top view of an autonomous cleaning robot in an environment.

FIGS. 7B and 7C illustrate an image captured by the autonomous cleaningrobot of FIG. 7A and a processed version of the image, respectively.

FIG. 8A is a top view of an autonomous cleaning robot in an environment.

FIG. 8B is a front view of a user computing device presenting a stitchedimage representation of the environment of FIG. 8A.

FIG. 9 is a flowchart of a method of controlling an autonomous cleaningrobot for navigating relative to an area rug.

FIGS. 10A-10C are top views of an autonomous cleaning robot in anenvironment with an area rug.

FIG. 11 is a flowchart of a method of maneuvering an autonomous cleaningrobot relative to a portion of a floor surface having an elevation lessthan an elevation of the robot.

FIG. 12 is a side view of an autonomous cleaning robot approaching aportion of a floor surface having an elevation less than an elevation ofthe robot.

FIG. 13 is a flowchart of a method of controlling a sensitivity of anautonomous cleaning robot for obstacle avoidance.

FIGS. 14A-14C are front views of a user computing device presenting auser interface for controlling a sensitivity of an autonomous cleaningrobot for obstacle avoidance.

FIG. 15 is a top view of an autonomous cleaning robot in an environmentin which a sensitivity of the robot for obstacle avoidance iscontrolled.

FIGS. 16A-16B are front views of a user computing device presenting anobject log and presenting a representation of an objected detected in anenvironment of an autonomous cleaning robot, respectively.

FIG. 17 is a side view of an autonomous cleaning robot approaching araised portion of a floor surface.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION

An autonomous mobile robot can be controlled to move about a floorsurface in an environment. In some implementations, the robot can beequipped with a camera that enables the robot to capture imagery of aportion of the floor surface ahead of the robot. As described herein,this imagery, alone or in combination with other sensor data produced bythe robot, can be used to create rich, detailed user-facingrepresentations of maps, and can also be used for controlling navigationof the robot relative to objects on the floor surface.

Example Autonomous Mobile Robots

Referring to FIG. 1, an autonomous mobile robot, e.g., an autonomouscleaning robot, 100 on a floor surface 10 in an environment 20, e.g., ahome, includes an image capture device 101 configured to capture imageryof the environment 20. In particular, the image capture device 101 ispositioned on a forward portion 122 of the robot 100. A field of view103 of the image capture device 101 covers at least a portion of thefloor surface 10 ahead of the robot 100. The image capture device 101can capture imagery of an object on the portion of the floor surface 10.

For example, as depicted in FIG. 1, the image capture device 101 cancapture imagery representing at least a portion of a rug 30 on the floorsurface 10. The imagery can be used by the robot 100 for navigatingabout the environment 20 and can, in particular, be used by the robot100 to navigate relative to the rug 30 so that the robot 100 avoidserror conditions that can potentially be triggered as the robot 100moves over the rug 30.

FIGS. 2 and 3A-3B depict an example of the robot 100. Referring to FIG.2, the robot 100 collects debris 105 from the floor surface 10 as therobot 100 traverses the floor surface 10. The robot 100 is usable toperform one or more cleaning missions in the environment 20 (shown inFIG. 1?) to clean the floor surface 10. A user can provide a command tothe robot 100 to initiate a cleaning mission. For example, the user canprovide a start command that causes the robot 100 to initiate thecleaning mission upon receiving the start command. In another example,the user can provide a schedule that causes the robot 100 to initiate acleaning mission at a scheduled time indicated in the schedule. Theschedule can include multiple scheduled times at which the robot 100initiates cleaning missions. In some implementations, between a startand an end of a single cleaning mission, the robot 100 may cease thecleaning mission to charge the robot 100, e.g., to charge an energystorage unit of the robot 100. The robot 100 can then resume thecleaning mission after the robot 100 is sufficiently charged. The robot100 can charge itself at a docking station. In some implementations, thedocking station can, in addition to charging the robot 100, evacuatedebris from the robot 100 when the robot 100 is docked at the dockingstation.

Referring to FIG. 3A, the robot 100 includes a housing infrastructure108. The housing infrastructure 108 can define the structural peripheryof the robot 100. In some examples, the housing infrastructure 108includes a chassis, cover, bottom plate, and bumper assembly. The robot100 is a household robot that has a small profile so that the robot 100can fit under furniture within a home. For example, a height H1 (shownin FIG. 2) of the robot 100 relative to the floor surface can be no morethan 13 centimeters. The robot 100 is also compact. An overall length L1(shown in FIG. 2) of the robot 100 and an overall width W1 (shown inFIG. 3A) are each between 30 and 60 centimeters, e.g., between 30 and 40centimeters, 40 and 50 centimeters, or 50 and 60 centimeters. Theoverall width W1 can correspond to a width of the housing infrastructure108 of the robot 100.

The robot 100 includes a drive system 110 including one or more drivewheels. The drive system 110 further includes one or more electricmotors including electrically driven portions forming part of theelectrical circuitry 106. The housing infrastructure 108 supports theelectrical circuitry 106, including at least a controller 109, withinthe robot 100.

The drive system 110 is operable to propel the robot 100 across thefloor surface 10. The robot 100 can be propelled in a forward drivedirection F or a rearward drive direction R. The robot 100 can also bepropelled such that the robot 100 turns in place or turns while movingin the forward drive direction F or the rearward drive direction R. Inthe example depicted in FIG. 3A, the robot 100 includes drive wheels 112extending through a bottom portion 113 of the housing infrastructure108. The drive wheels 112 are rotated by motors 114 to cause movement ofthe robot 100 along the floor surface 10. The robot 100 further includesa passive caster wheel 115 extending through the bottom portion 113 ofthe housing infrastructure 108. The caster wheel 115 is not powered.Together, the drive wheels 112 and the caster wheel 115 cooperate tosupport the housing infrastructure 108 above the floor surface 10. Forexample, the caster wheel 115 is disposed along a rearward portion 121of the housing infrastructure 108, and the drive wheels 112 are disposedforward of the caster wheel 115.

Referring to FIG. 3B, the robot 100 includes a forward portion 122 thatis substantially rectangular and a rearward portion 121 that issubstantially semicircular. The forward portion 122 includes sidesurfaces 150, 152, a forward surface 154, and corner surfaces 156, 158.The corner surfaces 156, 158 of the forward portion 122 connect the sidesurface 150, 152 to the forward surface 154.

In the example depicted in FIGS. 2, 3A, and 3B, the robot 100 is anautonomous mobile floor cleaning robot that includes a cleaning assembly116 (shown in FIG. 3A) operable to clean the floor surface 10. Forexample, the robot 100 is a vacuum cleaning robot in which the cleaningassembly 116 is operable to clean the floor surface 10 by ingestingdebris 105 (shown in FIG. 2) from the floor surface 10. The cleaningassembly 116 includes a cleaning inlet 117 through which debris iscollected by the robot 100. The cleaning inlet 117 is positioned forwardof a center of the robot 100, e.g., a center 162, and along the forwardportion 122 of the robot 100 between the side surfaces 150, 152 of theforward portion 122.

The cleaning assembly 116 includes one or more rotatable members drivenby a drive system, e.g., rotatable members 118 driven by a motor 120.The rotatable members 118 extend horizontally across the forward portion122 of the robot 100. The rotatable members 118 are positioned along aforward portion 122 of the housing infrastructure 108, and extend along75% to 95% of a width of the forward portion 122 of the housinginfrastructure 108, e.g., corresponding to an overall width W1 of therobot 100. Referring also to FIG. 2, the cleaning inlet 117 ispositioned between the rotatable members 118.

The rotatable members 118 are on a bottom portion of the robot 100, andare configured to rotate to direct debris into an interior of the robot100, e.g., into a debris bin 124 (shown in FIG. 2). As shown in FIG. 2,the rotatable members 118 are rollers that counter-rotate relative toone another. For example, the rotatable members 118 can be rotatableabout parallel horizontal axes 146, 148 (shown in FIG. 3A) to agitatedebris 105 on the floor surface 10 and direct the debris 105 toward thecleaning inlet 117, into the cleaning inlet 117, and into a suctionpathway 145 (shown in FIG. 2) in the robot 100. Referring back to FIG.3A, the rotatable members 118 can be positioned entirely within theforward portion 122 of the robot 100. The rotatable members 118 includeelastomeric shells that contact debris 105 on the floor surface 10 todirect debris 105 through the cleaning inlet 117 between the rotatablemembers 118 and into an interior of the robot 100, e.g., into the debrisbin 124 (shown in FIG. 2), as the rotatable members 118 rotate relativeto the housing infrastructure 108. The rotatable members 118 furthercontact the floor surface 10 to agitate debris 105 on the floor surface10.

The robot 100 further includes a vacuum system 119 operable to generatean airflow through the cleaning inlet 117 between the rotatable members118 and into the debris bin 124. The vacuum system 119 includes animpeller and a motor to rotate the impeller to generate the airflow. Thevacuum system 119 cooperates with the cleaning assembly 116 to drawdebris 105 from the floor surface 10 into the debris bin 124. In somecases, the airflow generated by the vacuum system 119 creates sufficientforce to draw debris 105 on the floor surface 10 upward through the gapbetween the rotatable members 118 into the debris bin 124. In somecases, the rotatable members 118 contact the floor surface 10 to agitatethe debris 105 on the floor surface 10, thereby allowing the debris 105to be more easily ingested by the airflow generated by the vacuum system119.

The robot 100 further includes a brush 126 that rotates about anon-horizontal axis, e.g., an axis forming an angle between 75 degreesand 90 degrees with the floor surface 10. The non-horizontal axis, forexample, forms an angle between 75 degrees and 90 degrees with thelongitudinal axes of the rotatable members 118. The robot 100 includes amotor 128 operably connected to the brush 126 to rotate the brush 126.

The brush 126 is a side brush laterally offset from a fore-aft axis FAof the robot 100 such that the brush 126 extends beyond an outerperimeter of the housing infrastructure 108 of the robot 100. Forexample, the brush 126 can extend beyond one of the side surfaces 150,152 of the robot 100 and can thereby be capable of engaging debris onportions of the floor surface 10 that the rotatable members 118typically cannot reach, e.g., portions of the floor surface 10 outsideof a portion of the floor surface 10 directly underneath the robot 100.The brush 126 is also forwardly offset from a lateral axis LA of therobot 100 such that the brush 126 also extends beyond the forwardsurface 154 of the housing infrastructure 108. As depicted in FIG. 3A,the brush 126 extends beyond the side surface 150, the corner surface156, and the forward surface 154 of the housing infrastructure 108. Insome implementations, a horizontal distance D1 that the brush 126extends beyond the side surface 150 is at least, for example, 0.2centimeters, e.g., at least 0.25 centimeters, at least 0.3 centimeters,at least 0.4 centimeters, at least 0.5 centimeters, at least 1centimeter, or more. The brush 126 is positioned to contact the floorsurface 10 during its rotation so that the brush 126 can easily engagethe debris 105 on the floor surface 10.

The brush 126 is rotatable about the non-horizontal axis in a mannerthat brushes debris on the floor surface 10 into a cleaning path of thecleaning assembly 116 as the robot 100 moves. For example, in examplesin which the robot 100 is moving in the forward drive direction F, thebrush 126 is rotatable in a clockwise direction (when viewed from aperspective above the robot 100) such that debris that the brush 126contacts moves toward the cleaning assembly and toward a portion of thefloor surface 10 in front of the cleaning assembly 116 in the forwarddrive direction F. As a result, as the robot 100 moves in the forwarddrive direction F, the cleaning inlet 117 of the robot 100 can collectthe debris swept by the brush 126. In examples in which the robot 100 ismoving in the rearward drive direction R, the brush 126 is rotatable ina counterclockwise direction (when viewed from a perspective above therobot 100) such that debris that the brush 126 contacts moves toward aportion of the floor surface 10 behind the cleaning assembly 116 in therearward drive direction R. As a result, as the robot 100 moves in therearward drive direction R, the cleaning inlet 117 of the robot 100 cancollect the debris swept by the brush 126.

The electrical circuitry 106 includes, in addition to the controller109, a memory storage element 144 and a sensor system with one or moreelectrical sensors, for example. The sensor system, as described herein,can generate a signal indicative of a current location of the robot 100,and can generate signals indicative of locations of the robot 100 as therobot 100 travels along the floor surface 10. The controller 109 isconfigured to execute instructions to perform one or more operations asdescribed herein. The memory storage element 144 is accessible by thecontroller 109 and disposed within the housing infrastructure 108. Theone or more electrical sensors are configured to detect features in anenvironment 20 of the robot 100. For example, referring to FIG. 3A, thesensor system includes cliff sensors 134 disposed along the bottomportion 113 of the housing infrastructure 108. Each of the cliff sensors134 is an optical sensor that can detect the presence or the absence ofan object below the optical sensor, such as the floor surface 10. Thecliff sensors 134 can thus detect obstacles such as drop-offs and cliffsbelow portions of the robot 100 where the cliff sensors 134 are disposedand redirect the robot accordingly.

The robot 100 can further include a wireless transceiver 149 (shown inFIG. 3A). The wireless transceiver 149 allows the robot 100 towirelessly communicate data with a communication network (e.g., thecommunication network 185 described herein with respect to FIG. 5). Therobot 100 can receive or transmit data using the wireless transceiver149, and can, for example, receive data representative of a map andtransmit data representative of mapping data collected by the robot 100.

Referring to FIG. 3B, the sensor system includes one or more proximitysensors that can detect objects along the floor surface 10 that are nearthe robot 100. For example, the sensor system can include proximitysensors 136 a, 136 b disposed proximate the forward surface 154 of thehousing infrastructure 108. Each of the proximity sensors 136 a, 136 bincludes an optical sensor facing outward from the forward surface 154of the housing infrastructure 108 and that can detect the presence orthe absence of an object in front of the optical sensor. For example,the detectable objects include obstacles such as furniture, walls,persons, and other objects in the environment 20 of the robot 100.

The sensor system includes a bumper system including the bumper 138 andone or more bump sensors that detect contact between the bumper 138 andobstacles in the environment 20. The bumper 138 forms part of thehousing infrastructure 108. For example, the bumper 138 can form theside surfaces 150, 152 as well as the forward surface 154. The sensorsystem, for example, can include the bump sensors 139 a, 139 b. The bumpsensors 139 a, 139 b can include break beam sensors, capacitive sensors,or other sensors that can detect contact between the robot 100, e.g.,the bumper 138, and objects in the environment 20. In someimplementations, the bump sensor 139 a can be used to detect movement ofthe bumper 138 along the fore-aft axis FA (shown in FIG. 3A) of therobot 100, and the bump sensor 139 b can be used to detect movement ofthe bumper 138 along the lateral axis LA (shown in FIG. 3A) of the robot100. The proximity sensors 136 a, 136 b can detect objects before therobot 100 contacts the objects, and the bump sensors 139 a, 139 b candetect objects that contact the bumper 138, e.g., in response to therobot 100 contacting the objects.

The sensor system includes one or more obstacle following sensors. Forexample, the robot 100 can include an obstacle following sensor 141along the side surface 150. The obstacle following sensor 141 includesan optical sensor facing outward from the side surface 150 of thehousing infrastructure 108 and that can detect the presence or theabsence of an object adjacent to the side surface 150 of the housinginfrastructure 108. The obstacle following sensor 141 can emit anoptical beam horizontally in a direction perpendicular to the forwarddrive direction F of the robot 100 and perpendicular to the side surface150 of the robot 100. For example, the detectable objects includeobstacles such as furniture, walls, persons, and other objects in theenvironment 20 of the robot 100. In some implementations, the sensorsystem can include an obstacle following sensor along the side surface152, and the obstacle following sensor can detect the presence or theabsence of an object adjacent to the side surface 152. The obstaclefollowing sensor 141 along the side surface 150 is a right obstaclefollowing sensor, and the obstacle following sensor along the sidesurface 152 is a left obstacle following sensor. The one or moreobstacle following sensors, including the obstacle following sensor 141,can also serve as obstacle avoidance sensors, e.g., similar to theproximity sensors described herein. In this regard, the left obstaclefollowing can be used to determine a distance between an object, e.g.,an obstacle surface, to the left of the robot 100 and the robot 100, andthe right obstacle following sensor can be used to determine a distancebetween an object, e.g., an obstacle surface, to the right of the robot100 and the robot 100.

In some implementations, at least some of the proximity sensors 136 a,136 b and the obstacle following sensor 141 each include an opticalemitter and an optical detector. The optical emitter emits an opticalbeam outward from the robot 100, e.g., outward in a horizontaldirection, and the optical detector detects a reflection of the opticalbeam that reflects off an object near the robot 100. The robot 100,e.g., using the controller 109, can determine a time of flight of theoptical beam and thereby determine a distance between the opticaldetector and the object, and hence a distance between the robot 100 andthe object. The sensor system further includes an image capture device140, e.g., a camera, directed toward a top portion 142 of the housinginfrastructure 108. The image capture device 140 generates digitalimagery of the environment 20 of the robot 100 as the robot 100 movesabout the floor surface 10. The image capture device 140 is angled in anupward direction, e.g., angled between 30 degrees and 80 degrees fromthe floor surface 10 about which the robot 100 navigates. The camera,when angled upward, can capture images of wall surfaces of theenvironment 20 so that features corresponding to objects on the wallsurfaces can be used for localization.

When the controller 109 causes the robot 100 to perform the mission, thecontroller 109 operates the motors 114 to drive the drive wheels 112 andpropel the robot 100 along the floor surface 10. In addition, thecontroller 109 operates the motor 120 to cause the rotatable members 118to rotate, operates the motor 128 to cause the brush 126 to rotate, andoperates the motor of the vacuum system 119 to generate the airflow. Tocause the robot 100 to perform various navigational and cleaningbehaviors, the controller 109 executes software stored on the memorystorage element 144 to cause the robot 100 to perform by operating thevarious motors of the robot 100. The controller 109 operates the variousmotors of the robot 100 to cause the robot 100 to perform the behaviors.

The sensor system can further include sensors for tracking a distancetraveled by the robot 100. For example, the sensor system can includeencoders associated with the motors 114 for the drive wheels 112, andthese encoders can track a distance that the robot 100 has traveled. Insome implementations, the sensor system includes an optical sensorfacing downward toward a floor surface. The optical sensor can be anoptical mouse sensor. For example, the optical sensor can be positionedto direct light through a bottom surface of the robot 100 toward thefloor surface 10. The optical sensor can detect reflections of the lightand can detect a distance traveled by the robot 100 based on changes infloor features as the robot 100 travels along the floor surface 10.

The controller 109 uses data collected by the sensors of the sensorsystem to control navigational behaviors of the robot 100 during themission. For example, the controller 109 uses the sensor data collectedby obstacle avoidance sensors of the robot 100, e.g., the cliff sensors134, the proximity sensors 136 a, 136 b and the bump sensors 139 a, 139b, to enable the robot 100 to avoid obstacles within the environment 20of the robot 100 during the mission.

The sensor data can be used by the controller 109 for simultaneouslocalization and mapping (SLAM) techniques in which the controller 109extracts features of the environment 20 represented by the sensor dataand constructs a map of the floor surface 10 of the environment 20. Thesensor data collected by the image capture device 140 can be used fortechniques such as vision-based SLAM (VSLAM) in which the controller 109extracts visual features corresponding to objects in the environment 20and constructs the map using these visual features. As the controller109 directs the robot 100 about the floor surface 10 during the mission,the controller 109 uses SLAM techniques to determine a location of therobot 100 within the map by detecting features represented in collectedsensor data and comparing the features to previously-stored features.The map formed from the sensor data can indicate locations oftraversable and nontraversable space within the environment 20. Forexample, locations of obstacles are indicated on the map asnontraversable space, and locations of open floor space are indicated onthe map as traversable space.

The sensor data collected by any of the sensors can be stored in thememory storage element 144. In addition, other data generated for theSLAM techniques, including mapping data forming the map, can be storedin the memory storage element 144. These data produced during themission can include persistent data that are produced during the missionand that are usable during a further mission. For example, the missioncan be a first mission, and the further mission can be a second missionoccurring after the first mission. In addition to storing the softwarefor causing the robot 100 to perform its behaviors, the memory storageelement 144 stores sensor data or data resulting from processing of thesensor data for access by the controller 109 from one mission to anothermission. For example, the map is a persistent map that is usable andupdateable by the controller 109 of the robot 100 from one mission toanother mission to navigate the robot 100 about the floor surface 10.

The persistent data, including the persistent map, enable the robot 100to efficiently clean the floor surface 10. For example, the persistentmap enables the controller 109 to direct the robot 100 toward open floorspace and to avoid nontraversable space. In addition, for subsequentmissions, the controller 109 is able to plan navigation of the robot 100through the environment 20 using the persistent map to optimize pathstaken during the missions.

The sensor system can further include a debris detection sensor 147 thatcan detect debris on the floor surface 10 of the environment 20. Thedebris detection sensor 147 can be used to detect portions of the floorsurface 10 in the environment 20 that are dirtier than other portions ofthe floor surface 10 in the environment 20. In some implementations, thedebris detection sensor 147 (shown in FIG. 2) is capable of detecting anamount of debris, or a rate of debris, passing through the suctionpathway 145. The debris detection sensor 147 can be an optical sensorconfigured to detect debris as it passes through the suction pathway145. Alternatively, the debris detection sensor 147 can be apiezoelectric sensor that detects debris as the debris impacts a wall ofthe suction pathway 145. In some implementations, the debris detectionsensor 147 detects debris before the debris is ingested by the robot 100into the suction pathway 145. The debris detection sensor 147 can be,for example, an image capture device that captures images of a portionof the floor surface 10 ahead of the robot 100. The controller 109 canthen use these images to detect the presence of debris on this portionof the floor surface 10.

The sensor system can further include the image capture device 101(shown in FIG. 3B). In some implementations, the sensor system includesa single image capture device corresponding to the image capture device101. In other words, in some implementations, the sensor system does notinclude the image capture device 140 but includes the image capturedevice 101. In implementations in which the sensor system includes asingle image capture device corresponding to the image capture device101, the imagery captured by the image capture device 101 can also beused for the functions described with respect to the image capturedevice 140. For example, the image capture device 101 can capture imagesof wall surfaces of the environment 20 so that features corresponding toobjects on the wall surfaces can be used for localization.

The image capture device 101 is positioned on the forward portion 122 ofthe robot 100 and is directed to capture imagery of at least a portionof the floor surface 10 forward of the robot 100. In particular, theimage capture device 101 can be directed in a forward direction F (shownin FIG. 3A) of the robot 100. The image capture device 101 can be, forexample, a camera or an optical sensor. Referring to FIG. 4, the fieldof view 103 of the image capture device 101 extends laterally andvertically. A center 160 of the field of view 103 can be, for example, 5to 45 degrees above the horizon or above the floor surface 10, e.g.,between 10 and 30 degrees, 10 and 40 degrees, 15 and 35 degrees, or 20and 30 degrees above the horizon or above the floor surface 10. Ahorizontal angle of view of the field of view 103 can be between 90 and150 degrees, e.g., between 100 and 140 degrees, 110 and 130 degrees, or115 and 125 degrees. A vertical angle of view of the field of view 103can be between 60 and 120 degrees, e.g., between 70 and 110 degrees, 80and 100 degrees, 85 and 95 degrees. The imagery can represent portionsof the floor surface 10 as well as other portions of the environment 20above the floor surface 10. For example, the imagery can representportions of wall surfaces and obstacles in the environment 20 above thefloor surface 10. As described herein, the image capture device 101 cangenerate imagery for producing representations of maps of theenvironment 20 and for controlling navigation of the robot 100 aboutobstacles.

Example Communication Networks

Referring to FIG. 5, an example communication network 185 is shown.Nodes of the communication network 185 include the robot 100, a mobiledevice 188, an autonomous mobile robot 190, and a cloud computing system192. Using the communication network 185, the robot 100, the mobiledevice 188, the robot 190, and the cloud computing system 192 cancommunicate with one another to transmit data to one another and receivedata from one another. In some implementations, the robot 100, the robot190, or both the robot 100 and the robot 190 communicate with the mobiledevice 188 through the cloud computing system 192. Alternatively oradditionally, the robot 100, the robot 190, or both the robot 100 andthe robot 190 communicate directly with the mobile device 188. Varioustypes and combinations of wireless networks (e.g., Bluetooth, radiofrequency, optical based, etc.) and network architectures (e.g., meshnetworks) may be employed by the communication network 185.

In some implementations, the mobile device 188 as shown in FIG. 5 is aremote device that can be linked to the cloud computing system 192 andcan enable the user to provide inputs on the mobile device 188. Themobile device 188 can include user input elements such as, for example,one or more of a touchscreen display, buttons, a microphone, a mouse, akeyboard, or other devices that respond to inputs provided by the user.The mobile device 188 alternatively or additionally includes immersivemedia (e.g., virtual reality) with which the user interacts to provide auser input. The mobile device 188, in these cases, is, for example, avirtual reality headset or a head-mounted display. The user can provideinputs corresponding to commands for the mobile robot 100. In suchcases, the mobile device 188 transmits a signal to the cloud computingsystem 192 to cause the cloud computing system 192 to transmit a commandsignal to the mobile robot 100. In some implementations, the mobiledevice 188 can present augmented reality images. In someimplementations, the mobile device 188 is a smartphone, a laptopcomputer, a tablet computing device, or other mobile device.

In some implementations, the communication network 185 can includeadditional nodes. For example, nodes of the communication network 185can include additional robots. Alternatively or additionally, nodes ofthe communication network 185 can include network-connected devices. Insome implementations, a network-connected device can generateinformation about the environment 20. The network-connected device caninclude one or more sensors to detect features in the environment 20,such as an acoustic sensor, an image capture system, or other sensorgenerating signals from which features can be extracted.Network-connected devices can include home cameras, smart sensors, andthe like. In the communication network 185 depicted in FIG. 5 and inother implementations of the communication network 185, the wirelesslinks may utilize various communication schemes, protocols, etc., suchas, for example, Bluetooth classes, Wi-Fi, Bluetooth-low-energy, alsoknown as BLE, 802.15.4, Worldwide Interoperability for Microwave Access(WiMAX), an infrared channel or satellite band. In some cases, thewireless links include any cellular network standards used tocommunicate among mobile devices, including, but not limited to,standards that qualify as 1G, 2G, 3G, or 4G. The network standards, ifutilized, qualify as, for example, one or more generations of mobiletelecommunication standards by fulfilling a specification or standardssuch as the specifications maintained by International TelecommunicationUnion. The 3G standards, if utilized, correspond to, for example, theInternational Mobile Telecommunications-2000 (IMT-2000) specification,and the 4G standards may correspond to the International MobileTelecommunications Advanced (IMT-Advanced) specification. Examples ofcellular network standards include AMPS, GSM, GPRS, UMTS, LTE, LTEAdvanced, Mobile WiMAX, and WiMAX-Advanced. Cellular network standardsmay use various channel access methods, e.g., FDMA, TDMA, CDMA, or SDMA.

Example Methods

Example methods are described below. These methods can be used toproduce user-facing representations of maps and for navigating anautonomous mobile robot in an environment. These methods can use imagerygenerated by a front-facing image capture device of an autonomous mobilerobot, e.g., the image capture device 101 of the robot 100.

The robot 100 can be controlled in certain manners in accordance withprocesses described herein. While some operations of these processes maybe described as being performed by the robot 100, by a user, by acomputing device, or by another actor, these operations may, in someimplementations, be performed by actors other than those described. Forexample, an operation performed by the robot 100 can be, in someimplementations, performed by the cloud computing system 192 or byanother computing device (or devices). In other examples, an operationperformed by the user can be performed by a computing device. In someimplementations, the cloud computing system 192 does not perform anyoperations. Rather, other computing devices perform the operationsdescribed as being performed by the cloud computing system 192, andthese computing devices can be in direct (or indirect) communicationwith one another and the robot 100. And in some implementations, therobot 100 can perform, in addition to the operations described as beingperformed by the robot 100, the operations described as being performedby the cloud computing system 192 or the mobile device 188. Othervariations are possible. Furthermore, while the methods, processes, andoperations described herein are described as including certainoperations or sub-operations, in other implementations, one or more ofthese operation or sub-operations may be omitted, or additionaloperations or sub-operations may be added.

Referring to FIG. 6, a method 600 is used for presenting arepresentation of a map of an environment, e.g., the environment 20, ona user-operable device, and for navigating an autonomous mobile robot,e.g., the robot 100, based on the map. The method 600 can include steps602, 604, 606, 608, 610, 612. The method 600 and its steps are describedwith respect to a robot 700 shown in FIG. 7A. The robot 700 is anautonomous cleaning robot similar to the robot 100, but in otherimplementations, other robots described herein may be used. Arepresentation of a map of an environment 702 of the robot 700 caninclude a stitched image representation of a floor surface 704 of theenvironment 702, with the stitched imagery representation beinggenerated by imagery captured by an image capture device of the robot700 (e.g., similar to the image capture device 101 described inconnection with the robot 100).

At the step 602, the robot 700 navigates about the floor surface 704while capturing imagery of the floor surface 704, e.g., using the imagecapture device. As described herein, the imagery captured by the imagecapture device can represent at least a portion of the floor surface704. The robot 700, as described herein, can navigate using sensor dataprovided by a sensor system of the robot 700, including imagery from theimage capture device of the robot 700. The robot 700 can navigate aboutthe floor surface 704 during a cleaning mission. For example, the robot700 can perform a vacuuming mission to operate a vacuum system of therobot 700 to vacuum debris on a floor surface of the environment 702.

At the step 604, the robot 700 transmits the imagery captured by itsimage capture device to a cloud computing system 650, e.g., similar tothe cloud computing system 192 described in connection with FIG. 5. Theimagery can be transmitted during a mission performed by the robot 100.Alternatively, the imagery can be transmitted at the end of the mission.At the step 606, the cloud computing system 650 receives the imageryfrom the robot 100.

At the step 608, a stitched image representation of the floor surface704 is produced based on the imagery obtained from the robot 700. Thestitched image representation corresponds to a top view of the floorsurface 704 that is produced from imagery captured by the image capturedevice of the robot 700, which is arranged to have a viewing directionparallel to the floor surface 704. As described herein, the imagecapture device 101 of the robot 100 (similar to the image capture deviceof the robot 700) is directed horizontally. As a result, imagery that iscaptured by the image capture device 101 represents a perspective view,from the side and from above, of the floor surface 10. In someimplementations, the stitched image representation of the floor surface704 is produced by the robot 700. The cloud computing system 650 can, insome cases, serve to store a copy of the stitched image representation,and to provide a communication channel between the mobile device 652 andthe robot 700.

Images representing perspective views of a floor surface can be stitchedtogether to form a top view of the floor surface. Referring to theexample depicted in FIG. 7A, in some implementations, because of theangle at which the image capture device of the robot 700 capturesimagery of the floor surface 704, only part of an image captured by theimage capture device can be used to form the top view of the floorsurface 704. Multiple images captured by the image capture device can bestitched together to form a top view of the floor surface 704 in theenvironment 702. The images can represent perspective views of differentportions of the floor surface 704 and can be combined to form a top viewrepresentation of a larger portion of the floor surface 704. Forexample, a first image can represent a perspective view of a firstportion of the floor surface 704, and a second image can represent aperspective view of a second portion of the floor surface 704. Theportions of the floor surface 704 represented in the first and secondimages can overlap. For example, the first portion of the floor surface704 and the second portion of the floor surface 704 can each include thesame portion of the floor surface 704.

Because the image capture device of the robot 700 is directed in theforward direction and has a perspective view of the floor surface 704,the imagery produced by the image capture device can represent aperspective view of a portion 706 of the floor surface 704 in theenvironment 702 that extends from a position in front of the robot 100up until an obstacle that occludes a view of the image capture device,such as a wall 705 in front of the robot 100. In this regard, the imagecapture device can, as described herein, detect objects and featuresforward of the robot 700.

While the portion 706 of the floor surface 704 represented in an imagecan be large because of the perspective view of the image capture deviceof the robot 700, only a smaller portion of the image is usable to formthe top view of the floor surface 704. FIG. 7B shows an example of animage 720 captured by the robot 700 in the environment 702. The image720 represents the floor surface 704, as well as a table 709, a chair710, and a rug 712 in the environment 702 (shown in FIG. 7A). The image720 represents the portion 706 of the floor surface 704. The image 720can be the originally captured image of the environment 702. In thisregard, the image 720 can be distorted due to the image capture devicehaving a wide field of view. FIG. 7C shows an example of an image 730produced from the image 720 through post-processing techniques to reducedistortion in an image. The image 730 presents an undistorted view ofthe environment 702. The image 730 can be used to form the stitchedimage representation of the floor surface 704. Only a portion 732 of theimage 730 is usable to form this stitched image representation. Theusable portion 732 corresponds to a portion 706 of the floor surface 704(shown in FIG. 7A) in front of the robot 100. Furthermore, the usableportion 732 corresponds to a portion 722 of the image 720 that isprocessed to form the usable portion 732. The portion of the image 720can be thin, e.g., three centimeters, two centimeters, one centimeter,one millimeter, or less. The portion of the image 720 can have a widthof a few pixels, e.g., 10 pixels, 8 pixels, 6 pixels, 4 pixels, 2pixels, or fewer. Because of the angle at which the image capture device101 is directed in the environment 20, e.g., directed along an axisparallel to the floor surface 10, the image 720 and hence the processedimage 730 provide a perspective view of the floor surface 704. By onlyusing a portion of the image 720 and the image 730, i.e., the usableportion 732 and the usable portion 722, for the stitched imagerepresentation, the stitched image representation can be representativeof a top view of the floor surface 704. Referring back to FIG. 7A, asthe robot 700 advances along the floor surface 704, the robot 700captures imagery of different portions of the floor surface 704,including the portion 706, and portions 714, 716, 718. Imagesrepresenting these portions 706, 714, 716, 718 can be stitched togetherto form the stitched image representation of a top view of the floorsurface 704.

Referring back to FIG. 6, after the step 608 in which the stitched imagerepresentation of the floor surface 704 is produced based on theimagery, at the step 610, the stitched image representation can be usedfor navigating the robot 700 or for presenting a representation of thefloor surface 704 to a user. For example, at the step 610, the robot 700maneuvers about the floor surface 704 based on the stitched imagerepresentation. The stitched image representation can be used incombination with sensor data collected by the robot 700 as the robot 700moves about the floor surface 704.

In some examples in which the robot 700 navigates about the floorsurface 704 using the stitched image representation, the robot 700 canuse the stitched image representation for determining locations ofobstacles and objects on the floor surface 704. For example, the robot700 could use the stitched image representation for determininglocations of the wall 705, the table 707, the chair 710, and the rug712. Based on the locations of these objects, the robot 700 can selectnavigational behaviors to navigate relative to these objects.

Referring to the example of FIG. 8A, an autonomous cleaning robot 800moves about an environment 802. The environment 802 includes multiplefloor surface types. For example, a floor surface 804 includes a portion806 having a first floor type, a portion 808 having a second floor type,a portion 810 having a third floor type, and a portion 812 having afourth floor type, a portion 814 have the third floor type, and aportion 816 having a fifth floor type. In the example depicted in FIG.8A, the first floor type is a hardwood surface, the second floor type isa mat surface, the third floor type is a carpet surface, the fourthfloor type is a tile surface, and the fifth floor type is a rug surface.In other implementations, the floor types can vary.

A stitched image representation can be formed in accordance withexamples described with respect to FIG. 6. For example, the robot 800can be similar to the robot 700, and capture imagery usable to form thestitched image representation. FIG. 8B illustrates an example of astitched image representation 818 that can be formed using imagerycaptured by the robot 800.

The stitched image representation 818 can be presented on a mobiledevice 820 (e.g., similar to the mobile device 652 described herein).The stitched image representation 818 includes information indicative ofthe floor surface types of the various portions of the environment 802.As described herein, the information in the stitched imagerepresentation 818 can correspond to processed imagery captured by therobot 800. Alternatively or additionally, the stitched imagerepresentation 818 can include computer generated images that representthe floor surface. The information indicative of the floor surface typesin the environment 802 can serve as reference points for a user todetermine what the stitched image representation 818 is depicting. Forexample, the user can use the floor surface types to distinguish betweendifferent rooms in the environment 802, and thus can easily determine alocation of the robot 800 in the environment 802 based on a location ofan indicator 822 of the location of the robot 800 overlaid on thestitched image representation 818.

Other objects can be identified from the stitched image representation.Examples of navigation of autonomous cleaning robots relative to arearugs are described herein with respect to at least FIGS. 9 and 10A-10C.In further examples, obstacles that would, for example, trigger obstacleavoidance behavior of the robot 700, can include cords, chairs,bedframes, and desks.

The objects can further include debris on the floor surface 10. Forexample, the imagery can represent debris on a portion of the floorsurface 10 that can be cleaned by the robot 700 by maneuvering over theportion of the floor surface 10.

In some implementations, the stitched image representation can betransmitted to other autonomous mobile robots that operate in theenvironment 702. The other autonomous mobile robots can use the stitchedimage representation for navigating about the environment 702. In someimplementations, the stitched image representation can be used incombination with sensor data collected by these other autonomous mobilerobots. These other autonomous mobile robots can include one or moreautonomous cleaning robots, e.g., a vacuum cleaning robot, a moppingrobot, or other autonomous cleaning robots. In some implementations, anobject represented in the stitched image representation can correspondto an obstacle for the robot 700 but can correspond to debris that iscleanable by another autonomous cleaning robot. For example, if therobot 700 is a vacuum cleaning robot, and another autonomous cleaningrobot operating in the environment 702 is a mopping cleaning robot, apuddle in the environment 702 can correspond to an obstacle for therobot 700 and can correspond to cleanable debris for the otherautonomous cleaning robot. Other autonomous mobile robots are possible.For example, the autonomous mobile robots can include one or moreautonomous patrol robots.

In some implementations, at the step 612, a representation of thestitched image representation of the floor surface 704 is presented tothe user. For example, a mobile device 652, e.g., similar to the mobiledevice 188, can present the representation of the stitched imagerepresentation, thereby providing the user with a top viewrepresentation of the floor surface 704. The top view representation cancorrespond to the stitched image representation and can indicate thefloor types through the portions of the imagery that represent the floorsurface 704. Alternatively, floor types can be identified from thestitched image representation, and the mobile device 652 can presentindicators (e.g., images/textures/backgrounds) based on these floortypes. In some implementations, for privacy, the representationpresented on the mobile device 652 can include stock images orcomputer-generated images indicative of the identified floor types toindicate the floor types.

The stitched image representation of the floor surface 704 can bepresented with a representation of other objects and features in theenvironment 20. For example, as described herein, indicators ofobstacles on the floor surface 704 can be overlaid on the stitched imagerepresentation of the floor surface 704. For example, referring brieflyback to the example of FIG. 7A, the mobile device 652 could presentindicators of the table 709, the chair 710, and the rug 712. Because theimagery captured by the robot 700 is captured using ahorizontally-directed camera, the imagery can also represent portions ofwalls in the environment 20. Referring briefly back to the example ofFIG. 7A, the mobile device 652 could present indicators of the wall 705and objects on the wall 705, such as windows, paintings, photographs,and other objects on the wall 705. The user can interact with the mobiledevice 652 to switch between different views of the environment 20. Forexample, the stitched image representation can be presented to show thetop view representation of the floor surface 704. The user can interactwith the mobile device 652 to switch to a side view in the environment20 in which the mobile device 652 presents a representation of walls inthe environment 20. The side view can also be produced using the imagerycaptured by the image capture device 101 of the robot 700. Thesedifferent views can include the top view representation of the floorsurface 704 as well as the side view representations of the environment20, and these views can be combined for presenting a three-dimensionalrepresentation of the environment 20. This three-dimensionalrepresentation can represent both the floor surface 704 (a horizontalplane) and walls in the environment (vertical planes). Thisrepresentation can be formed from multiple images captured by the robot700 at different locations in the environment 20. These images, becausethey are captured are different locations in the environment 20, canprovide a stereoscopic view of a portion of the environment 20.

Objects and obstacles represented in these images as described hereincan be overlaid on this three-dimensional representation, therebyaccurately depicting placement of doors and windows in the environment20. Alternatively or additionally, machine learning techniques can beused to detect distances of objects from the robot 700 that appear in asingle image captured by the robot 700. Based on these distances, athree-dimensional representation can be generated to be presented to theuser on the mobile device 652.

Referring to FIG. 9, a method 900 is performed by an autonomous mobilerobot, e.g., the robot 100, to navigate the robot relative to an arearug in an environment. The method 900 can include steps 902, 904, and906. The method 900 and its steps are described with respect to a robot1000 shown in FIGS. 10A-10C. The robot 1000 is an autonomous cleaningrobot similar to the robot 100, but in other implementations, otherrobots may be used.

At the step 902, referring also to FIG. 10A, the robot 1000 navigatesabout a floor surface 1004 of an environment 1002 while capturingimagery of the floor surface 1004, e.g., using an image capture deviceof the robot 1000 similar to the image capture device 101 of the robot100 as described herein. The imagery captured by the image capturedevice can represent at least a portion of the floor surface 1004. Therobot 1000, as described herein, can navigate using sensor data providedby a sensor system of the robot 1000, including imagery from the imagecapture device of the robot 1000. The robot 1000 can navigate about thefloor surface 1004 during a cleaning mission. For example, the robot1000 can perform a vacuuming mission to operate a vacuum system of therobot to vacuum debris on a floor surface of the environment 1002.

At the step 904, the robot 1000 detects a rug 1006 on a portion of thefloor surface 1004 based on the imagery captured by the image capturedevice. The robot 1000 can detect the rug 1006 before moving over therug 1006. By using the horizontally directed image capture device, therobot 1000 can detect objects and features ahead of the robot 1000 andcan, in particular, detect the rug 1006.

At the step 906, after detecting the rug 1006, the robot 1000 maneuversonto the rug 1006 along a path 1008 selected based on the imagerycaptured by the image capture device. The path 1008 can be selected toreduce a likelihood that the robot 1000 encounter an error condition asthe robot 1000 moves from off of the rug 1006 and then onto the rug1006. The imagery captured by the robot 1000 can be analyzed to identifythe path 1008 to reduce the likelihood of an error condition. Theimagery can include a plurality of images that are stitched together toform a stitched image representation produced like the way the stitchedimage representation described with respect to FIG. 6 is produced. Thepath 1008 can be determined based on the stitched image representation.Alternatively or additionally, the path 1008 can be determined based onthe imagery captured by the robot 1000, including portions of theimagery that may not be formed into the stitched image representation.The imagery used by the robot 1000 to select the path 1008 could includemultiple images captured by the robot 1000. The imagery or the stitchedimage representation can be indicative of locations of the cornerportions 1010 and/or the tassels 1012 of the rug 1006. The imagery orthe stitched image representation can also be indicative of a locationof the rug 1006 relative to the robot 1000, a shape of the rug 1006, orother geometry of the rug 1006 can determined. The path 1008 can beselected such that the robot 1000 avoids moving over the corner portions1010, or moving over the tassels 1012.

The error condition could be a stasis condition of a component of therobot 1000 in which a movable component of the robot 1000 is unable tomove due to, for example, an object entrained in the movable component.The error condition could be, for example, a stasis condition of arotatable member of the robot 1000 (e.g., similar to one of therotatable members 118 of the robot 100), a stasis condition for a drivewheel of the robot 1000 (e.g., similar to one of the drive wheels 112 ofthe robot 100), or a stasis condition for a caster wheel of the robot1000 (e.g., similar to the caster wheel 115 of the robot 100). A stasiscondition for a movable component of the robot 1000 could occur as therobot 1000 moves from off of the rug 1006 to onto the rug 1006 if aportion of the rug 1006 impedes movement of the movable component.

For example, certain geometries of the rug 1006 can become entrained inthe rotatable members, the drive wheels, or the caster wheel of therobot 1000. In the example depicted in FIG. 10A, one of the cornerportions 1010 of the rug 1006 can become entrained in the rotatablemembers, the drive wheels, or the caster wheel of the robot 1000. Inaddition, tassels 1012 of the rug 1006 can become entrained in one ofthese components.

To avoid the corner portions 1010 causing an error condition, the path1008 onto the rug 1006 can be selected such that the robot 1000 avoidsmoving over the corner portions 1010. The corner portions 1010correspond to locations on the rug 1006 where two edges, e.g., an edge1014 and an edge 1016, meet one another at an angle. These cornerportions 1010 can be susceptible to being entrained in a movablecomponent of the robot 1000. The path 1008 can be selected such that afootprint of the robot 1000 does not move over any of the cornerportions 1010 as the robot 1000 moves onto the rug 1006. In addition,the path 1008 can be selected such that a cleaning path, e.g., a pathcovered by the rotatable members of the robot 1000, does not extend overthe corner portions 1010 as the robot 1000 moves onto the rug 1006,thereby reducing a risk that the robot 1000 ingests part of the cornerportions 1010. To avoid the tassels 1012 causing an error condition, thepath 1008 onto the rug 1006 can be selected such that the robot 1000avoids moving over the tassels 1012. The tassels 1012 of the rug 1006can be thin and elongate fabric that can easily bend when the robot 1000moves over the tassels 1012. Bases of the tassels 1012 are attached tothe edge 1014 of the rug 1006, and the tassels 1012 extend across thefloor surface 1004 from the edge 1014 outwardly away from a centralportion of the rug 1006. The tassels 1012 can bend in response tofriction between a bottom portion of the robot 1000 and the tassels1012, and can, in some cases, be easily entrained by the rotatablemembers, the drive wheels, or the caster wheel of the robot 1000. Toavoid a stasis condition for any of these components, in someimplementations, when the robot 1000 moves from a location off of therug 1006 to a location on the rug 1006, the robot 1000 moves across theedge 1016 of the rug 1006 that does not include the tassels 1012. Therobot 1000 can avoid moving onto the rug 1006 across an edge 1016 thatincludes the tassels 1012 such that the tassels 1012 do not becomeentrained in movable components of the robot 1000.

In the example shown in FIG. 10A, the path 1008 includes a portion 1008a substantially perpendicular to the edge 1016. An angle between theportion 1008 a of the path 1008 and the edge 1016 can be no more than 1to 10 degrees, e.g., no more than 1 to 7 degrees, 1 to 5 degrees, or 1to 3 degrees. In some implementations, the portion 1008 a of the path1008 intersects with the edge 1016 at a steeper angle, e.g., more than 1to degrees. By moving along the portion 1008 a of the path 1008, therobot 1000 can avoid stasis conditions caused by the rug 1006.

After the robot 1000 moves along the path 1008, referring to FIG. 10B,after the robot 1000 moves along a path 1018 to clean the rug 1006. Inparticular, the robot 1000 moves in a manner to cover a surface of therug 1006 without moving off of the rug 1006. The path 1018 can have, forexample, a cornrow pattern in which the robot 1000 moves alongsubstantially parallel rows across the rug 1006 to clean the rug 1006.The path 1018 can be determined using methods similar to those used fordetermining the path 1008 of FIG. 10A, e.g., using the imagery capturedby the robot 1000 or the stitched image representation produced from theimagery.

Referring to FIG. 10C, after the robot 1000 has cleaned the rug 1006,the robot 1000 can move along a path 1020 to move from a location on therug 1006 to a location off of the rug 1006. The path 1020 can beselected to avoid error conditions triggered by the tassels 1012 or thecorner portions 1010. In the example depicted in FIG. 10C, the path 1020includes a portion 1020 a that is perpendicular to the edge 1016,thereby allowing the robot 1000 to avoid encountering the tassels 1012or the corner portions 1010. In some implementations, a region proximateto the tassels 1012 requires cleaning, and the robot 1000 may need tomove over the tassels 1012 in order to clean this region. Rather thancleaning this region by moving from a location off of the rug 1006 to alocation on the rug 1006, the robot 1000 can be maneuvered to move overthe tassels 1012 only in a way that reduces a risk that the tassels 1012are entrained in the movable components of the robot 1000. For example,the robot 1000 can be controlled to move along a path in which the robot1000 only moves over the tassels 1012 if the robot 1000 is moving from alocation on the rug 1006 to a location off of the rug 1006. In someimplementations, the path 1020 is selected such that the robot 1000moves over the tassels 1012 in a direction that is substantiallyparallel to a direction that the tassels 1012 extend from the edge 1016.For example, the direction that the robot 1000 moves and the directionthat the tassels 1012 extend from the edge form an angle that is no morethan 1 to 10 degrees, e.g., no more than 1 to 7 degrees, 1 to 5 degrees,or 1 to 3 degrees. If the robot 1000 moves over the tassels 1012 in thismanner, the tassels 1012 tend not to bend and are less likely to becomeentrained in movable components of the robot 1000. The path 1020 can bedetermined using methods similar to those used for determining the path1008 of FIG. 10A, e.g., using the imagery captured by the robot 1000 orthe stitched image representation produced from the imagery.

Referring to FIG. 11, a method 1100 is performed by an autonomous mobilerobot, e.g., the robot 100, to navigate the robot relative to a lowerportion of the floor surface that has a lower elevation than an upperportion of the floor surface along which the robot is moving. The lowerportion and the upper portion of the floor surface form a cliff that therobot can avoid moving over in order to prevent fall damage to therobot. The method 1100 includes steps 1102, 1104, 1106, and 1108. Themethod 1100 and its steps are described with respect to a robot 1200shown in FIG. 12. The robot 1200 is an autonomous cleaning robot similarto the robot 100, but in other implementations, other robots may beused. As described herein, the method 1100 allows the robot 1200 todetect a cliff before a cliff sensor of the robot 1200 can detect thecliff, and thus allows the robot 1200 to respond sooner to the presenceof the cliff.

In the step 1102, referring also to FIG. 12, the robot 1200 navigatesabout a floor surface 1204 of an environment 1202 at a first speed whilecapturing imagery of the floor surface 1204, e.g., using an imagecapture device of the robot 1200 similar to the image capture device 101of the robot 100 as described herein. The imagery captured by the imagecapture device can represent at least a portion of the floor surface1204. The robot 1200, as described herein, can navigate using sensordata provided by a sensor system of the robot 1200, including imageryfrom the image capture device of the robot 1200. The robot 1200 cannavigate about the floor surface 1204 during a cleaning mission. Forexample, the robot 1200 can perform a vacuuming mission to operate avacuum system of the robot to vacuum debris on a floor surface of theenvironment 1002. The robot 1200 can move at the first speed from point1206 a to point 1206 b. The first speed can correspond to a speed atwhich the robot 1200 moves across the floor surface 1204 in a coveragebehavior or an obstacle following behavior. A speed of the robot 1200can be reduced relative to the first speed in response to detection of afeature or object and initiation of a behavior to avoid the feature orobject. For example, the robot 1200 may reduce its speed as the robot1200 approaches the feature or object to avoid contacting the feature orobject. The method 1100 illustrates an example in which the feature is acliff.

At the step 1104, the robot 1200 detects a cliff based on the imagerycaptured at the step 1102. The cliff can correspond to a reduction inelevation of the floor surface 1204. For example, the robot 1200 ispositioned on a first portion 1204 a of the floor surface 1204 that isat a higher elevation than a second portion 1204 b of the floor surface1204. The first and second portions 1204 a, 1204 b form a cliff 1208.The robot 1200 could detect the cliff 1208 at, for example, the point1206 b. The cliff 1208 could be identified from the captured imagery. Insome implementations, the cliff 1208 is identified in a portion of thecaptured imagery that is beyond a portion used for forming a stitchedimage representation. In some implementations, the cliff 1208 isidentified in a portion of the captured imagery that is used to form thestitched image representation.

At the step 1106, the robot 1200 is maneuvered relative to the cliff1208 at a second speed that is less than the first speed that the robot1200 moved between the point 1206 a and the point 1206 b. For example,at the point 1206 b, upon detecting the cliff 1208, the robot 1200reduces its speed. The robot 1200 can reduce its speed from the firstspeed to the second speed and can make this speed reduction before therobot 1200 detects the cliff 1208 using a cliff sensor of the robot 1200(e.g., similar to one of the cliff sensors 134 of the robot 100 asdescribed herein).

In some implementations, at the step 1104, the robot 1200 can detect thecliff 1208, and at the step 1106, the robot 1200 reduces its speed onlyafter the robot 1200 is within a distance from the cliff 1208. Thedistance between the robot 1200 and the cliff 1208 can be determinedbased on the captured imagery. The distance between the point 1206 b andthe cliff 1208 can be between 0.1 and 1 meter from the cliff 1208, e.g.,between 0.1 and 0.7, 0.1 and 0.5, or 0.1 and 0.3 meters from the cliff1208. The distance can be between 50% to 300% of a length of the robot1200, e.g., between 50% and 250%, between 50% and 200%, or between 50%and 150% of the length of the robot 1200. The robot 1200 can initiatereduction to the second speed based on determining, from the imagerycaptured by the image capture device, the robot 1200 is no more than thedistance from the cliff 1208.

At the step 1108, the robot 1200 detects the cliff 1208 using the cliffsensor of the robot 1200. The robot 1200 can detect the cliff 1208 whena portion of the robot 1200 is moved over the second portion 1204 b ofthe floor surface 1204, thereby allowing the cliff sensor of the robot1200 to detect an absence of an object below the portion of the robot1200. Upon detecting the cliff 1208, the robot 1200 is maneuvered alongthe first portion 1204 a of the floor surface 1204 away from the secondportion 1204 b of the floor surface, i.e., away from the cliff 1208. Therobot 1200 can turn such that the robot 1200 moves away from the cliff1208 or such that the robot 1200 moves along the cliff 1208.

Referring to FIG. 13, a method 1300 is used for controlling an obstacleavoidance sensitivity of a robot, e.g., the robot 100. As describedherein, an autonomous mobile robot can include a sensor system with oneor more electrical sensors. The sensors can be used to detect variousobjects and features in the environment, and these sensors, upondetecting an object or feature, can trigger avoidance behavior in whichthe robot avoids the object or feature. For example, the sensor can be acliff sensor (e.g., one of the cliff sensors 134 of the robot 100), aproximity sensor (e.g., one of the proximity sensors 136 a, 136 b of therobot 100), or an image capture device (e.g., the image capture device101 of the robot 100). The sensor can detect a particular feature (e.g.,an obstacle such as a cliff, a wall, or other feature), and then therobot can be maneuvered to avoid the feature in response to the robotbeing sufficiently close to the feature. In some implementations, therobot initiates an obstacle avoidance behavior. In the obstacleavoidance behavior, the robot can reduce its speed when the robot iswithin a first distance from the obstacle, and then can turn away fromthe obstacle to avoid the obstacle when the robot is within a seconddistance from the obstacle.

As described herein with respect to the method 1300, in someimplementations, an obstacle avoidance sensitivity for the robot can beset, and the first distance and the second distance can vary dependingon the set obstacle avoidance sensitivity. The method 1300 includessteps 1302, 1304, 1306, 1308, 1310, 1312, 1314, 1316, 1318, 1320. Themethod 1300 is described in connection with FIGS. 14A-14C showing amobile device 1350 being used to set the obstacle avoidance sensitivityat different levels, and in connection with FIG. 15 showing a robot 1500moving along a floor surface 1504 in an environment 1502 with anobstacle 1506.

At the step 1302, referring also to FIG. 15, the robot 1500 navigatesabout the floor surface 1504 while capturing imagery of the floorsurface 1504, e.g., using an image capture device of the robot 1500similar to the image capture device 101 of the robot 100 as describedherein. The imagery captured by the image capture device can representat least a portion of the floor surface 1504. The robot 1500, asdescribed herein, can navigate using sensor data provided by a sensorsystem of the robot 1500, including imagery from the image capturedevice of the robot 1500. The robot 1500 can navigate about the floorsurface 1504 during a cleaning mission. For example, the robot 1500 canperform a vacuuming mission to operate a vacuum system of the robot tovacuum debris on a floor surface of the environment 1502. During themission, the robot 1500 can detect an obstacle 1506 using the sensorsystem. The robot 1500 can detect the obstacle 1506 using the imagecapture device of the robot 1500, and/or using a bump sensor or aproximity sensor of the robot 1500.

At the step 1304, the robot 1500 transmits the captured imagery to acloud computing system 1352, e.g., similar to the cloud computing system192 described in connection with FIG. 5. The imagery can be transmittedduring a mission performed by the robot 1500. Alternatively, the imagerycan be transmitted at the end of the mission. At the step 1306, thecloud computing system 650 receives the imagery from the robot 1500.

At the step 1308, obstacles, rooms, and/or feature in the environment1502 are identified. For example, the cloud computing system 1352 canidentify the obstacle 1506, as well as rooms such as a room 1508 and aroom 1510 in the environment 1502. The cloud computing system 1352 canidentify locations of obstacles detected in the environment 1502, andcan identify relative positions of rooms in the environment 1502. At thestep 1310, a stitched image representation of the floor surface 1504 canbe produced based on the imagery captured by the robot 1500. Thestitched image representation can be produced like that described withrespect to the step 608 in connection with FIG. 6.

At the step 1312, the cloud computing system 1352 transmits dataindicative of the identified obstacles, rooms, and/or features in theenvironment 1502 and data indicative of the stitched imagerepresentation. At the step 1314, the mobile device 1350 receives thedata indicative of the identified obstacles, rooms, and/or features inthe environment 1502 and the data indicative of the stitched imagerepresentation. In some implementations, the steps 1308 and 1310 areperformed by the robot 1500 rather than the cloud computing system 1352.

At the step 1316, referring also to FIGS. 14A-14C, the mobile device1350 receives a user input indicative of a user-selected obstacleavoidance sensitivity. In the example shown in FIGS. 14A-14C, the mobiledevice 1350 is a mobile phone in which a user input element of themobile device 1350 is a touchscreen, and a user output element is adisplay. The user can operate the touchscreen to provide a user inputindicative of a user-selected obstacle avoidance sensitivity. FIGS.14A-14C depict a low sensitivity selection 1402, a medium sensitivityselection 1404, and a high sensitivity selection 1406. In someimplementations, user selections can be indicative of obstacle avoidancesensitivities for a particular obstacle. In the example depicted inFIGS. 14A-14C, the selections 1402, 1404, 1406 are indicative ofobstacle avoidance sensitivities to the obstacle 1506.

The mobile device 1350 can present an indicator 1408 representing theobstacle 1506 (shown in FIG. 15), and can further present an indicator1410 of a range of available sensitivities that the user could select.The indicator 1408 can be presented based on the imagery captured by therobot 1500. The indicator 1408 can include a representation of theobstacle 1506, and can also include a top view representation of thefloor surface 1504. The indicator 1408 can a graphical representation ofa top view of the environment 1502 and indicate a location of theobstacle 1506 in the environment 1502. The indicator 1408 can furthervisually represent the sensitivities for the corresponding selections1402, 1404, 1406. For example, the indicator 1408 can visually representthese sensitivities by indicating different-sized zones around theobstacle 1506. In some implementations, the indicator 1408 can provide arepresentation of an image captured by the robot 1500, with the imagerepresenting at least a portion of the obstacle 1506.

A user can interact with the indicator 1410 to provide the selections1402, 1404, 1406. For example, the indicator 1410 can represent a sliderthat the user can interact with to provide the selections 1402, 1404,1406. In some implementations, the indicator 1410 can include a list ofsensitivity levels, with the levels being selectable by the user toprovide the selections 1402, 1404, 1406.

In some implementations, rather than being indicative of obstacleavoidance sensitivities for a particular obstacle, user selections canbe indicative of obstacle avoidance sensitivities for a room. Forexample, the mobile device 1350 can present an indicator of a room,e.g., one of the rooms 1508, 1510, and provide an indicator of a rangeof available obstacle avoidance sensitivities that the user could selectfor the room. The user-selected obstacle avoidance sensitivity cancorrespond to a sensitivity to obstacles detected in the room. The usercan interact with the mobile device to provide a selection indicative ofuser-selected obstacle avoidance sensitivity to obstacles in the room.In further implementations, user selections can be indicative ofobstacle avoidance sensitivities for the environment 1502 as a whole.For example, a user-selected obstacle avoidance sensitivity cancorrespond to a sensitivity to obstacles detected in the environment1502.

At the step 1318, the robot 1500 maneuvers about the floor surface 1504.The robot 1500 can maneuver about the floor surface 1504 during amission of the robot 1500. This mission can be subsequent to the missionperformed for the step 1302. At the step 1320, the robot 1500 initiatesan avoidance behavior to avoid the obstacle 1506 based on theuser-selected obstacle avoidance sensitivity. As the robot 1500 movesabout the floor surface 1504, the robot 1500 can initiate the obstacleavoidance behavior to avoid the obstacle 1506 in response to detectingthe obstacle 1506. The obstacle avoidance behavior can be initiatedbased on the user-selected obstacle avoidance sensitivity. In someimplementations, the user-selected obstacle avoidance sensitivity canindicate a threshold for a distance between the robot 1500 and theobstacle 1506 at which the robot 1500 would initiate the obstacleavoidance behavior. For example, as depicted in FIG. 15, distancethresholds 1512, 1514, 1516 correspond to the selections 1402, 1404,1406, respectively. The robot 1500 initiates the obstacle avoidancebehavior based on a distance between the obstacle 1506 and the robot1500 being no more than the distance threshold 1512, 1514, 1516. Theselections 1402, 1404, 1406, referring briefly back to FIGS. 14A-14C,can be user selections of distances. The selection 1402 can correspondto a distance between 0 and 15 centimeters, e.g., between 1 and 5centimeters, between 1 and 10 centimeters, between 1 and 15 centimeters,less than 1 centimeter, at least 1 centimeter, at least 3 centimeters,at least 5 centimeters, etc. The selection 1404 can correspond to adistance between 3 and 30 centimeters, e.g., between 5 and 15centimeters, between 5 and 20 centimeters, between 5 and 25 centimeters,at least 3 centimeters, at least 5 centimeters, at least 7 centimeters,or at least 10 centimeters, etc. The selection 1406 can correspond to adistance between 5 and 60 centimeters, e.g., between 10 and 30centimeters, between 20 and 40 centimeters, between 30 and 50centimeters, between 40 and 60 centimeters, at least 5 centimeters, atleast 7 centimeters, at least 10 centimeters, etc.

In some implementations, the user-selected obstacle avoidancesensitivity represents a likelihood threshold that the obstacle 1506 ispresent on a portion of the floor surface 1504. As the robot 1500 movesabout the floor surface 1504, the robot 1500 can determine a likelihoodthat the obstacle 1506 is proximate to the robot 1500, or is ahead ofthe robot 1500. The likelihood can be determined based on sensor datafrom the current mission that the robot 1500 is performing, as well asbased on sensor data from one or more previously performed missions. Forexample, the obstacle 1506 can be detected in a previously performedmission, such as the mission described with respect to the step 1302. Inaddition, the likelihood can be determined based on a mobility of theobstacle 1506. For example, the obstacle 1506 can have a high mobility,such as a cord, clothing, or other obstacle that is likely to be pickedup by a user and placed elsewhere or removed from the floor surface1504. If the obstacle 1506 has high mobility and is detected in a firstmission, the likelihood that the obstacle 1506 is present in a secondmission could be low. The obstacle 1506, alternatively, can have a lowmobility, such as a table or a couch. If the obstacle 1506 has lowmobility and is detected in a first mission, the likelihood that theobstacle 1506 is present in a second mission could be high.

In some implementations, rather than being user-selected sensitivity,the sensitivity can be automatically selected, for example, by the robot1500 or the cloud computing system 1352. The sensitivity to an obstaclecan be selected based on whether the robot 1500, in one or more previousmissions, experienced an error condition near the obstacle. After therobot 1500 has initially detected the obstacle, subsequent missions inwhich the robot 1500 does not detect the obstacle can reduce thesensitivity of the robot 1500 to the obstacle. In some implementations,the indicator 1410 can indicate the automatically-selected sensitivity,and then the user can interact with the indicator 1410 to change thesensitivity.

Additional Alternative Implementations

A number of implementations have been described. Other implementationsare possible. While some implementations are described with respect to asingle autonomous mobile robot, e.g., the robot 100, the robot 700, therobot 1000, the robot 1200, and the robot 1500, in some implementations,data from multiple autonomous mobile robots operating in the environmentcan be used. For example, the imagery captured by the robot 100 can beused in combination with sensor data generated by the robot 190described with respect to FIG. 5 to form a user-facing map or to controlnavigation of the robot 100 or the robot 190. In some implementations,the robot 190 can also have a front facing image capture device similarto the image capture device 101 of the robot 100. The image capturedevice of the robot 190 can capture imagery that can be used incombination with the imagery captured by the robot 100 to generate astitched image representation of a floor surface.

The image capture device 101, as described herein, can be a single imagecapture device of the robot 100. In some implementations, the robot 100can include two or more front-facing image capture devices, and imageryfrom the two or more front-facing image capture devices can be used forthe methods described herein.

The image capture device 101, as described herein, can be horizontallydirected in the forward direction F of the robot 100. In someimplementations, the image capture device 101 is angled relative to ahorizontal axis. For example, the image capture device 101 can be angleddownward at an angle between 5 and 30 degrees, e.g., between 5 and 25degrees, 5 and 20 degrees, or 5 and 15 degrees.

The method 900 depicted in FIG. 9 is described in connection with therug 1006 depicted in FIGS. 10A-10C. The method 900 can be used with rugshaving other geometries. For example, non-rectangular rugs could includemultiple protruding portions. In some cases, a rug could includenon-linear geometry along an edge due to a rip. The method 900 could beimplemented to avoid the portion including the non-linear geometry.

Referring to FIGS. 10A-10C, the robot 1000 is described as avoidingmoving onto the rug 1006 across the edge 1016 that includes the tassels1012 such that the tassels 1012 do not become entrained in movablecomponents of the robot 1000. In some implementations, the robot 1000can move along a path that moves over the tassels 1012. In suchimplementations, the robot 1000 can reduce a speed of rotation of therotatable members of the robot 1000 as the robot 1000 moves over thetassels 1012. The robot 1000 can reduce a speed of rotation of therotatable members when the robot 1000 is at a location off of the rug1006 and before the robot 1000 moves onto the rug 1006. In this regard,the robot 1000 can rotate the rotatable member at a first speed ofrotation as the robot 1000 moves about a portion of the floor surface1004 off of the rug 1006, and then rotate the rotatable member a secondspeed of rotation as the robot 1000 moves from the portion of the floorsurface 1004 off of the rug 1006 to a portion of the floor surface 1004on the rug 1006, with the second speed of rotation being less than thefirst speed of rotation. In some implementations, the robot 1000 reducesthe speed of rotation by deactivating a drive system that drives therotatable members of the robot 1000 such that the rotatable members nolonger rotate. In some implementations, the robot 1000 reduces an amountof power delivered to the drive system for the rotatable members.

After the robot 1000 is on the rug 1006 and is beyond the edges of therug 1006, the robot 1000 can increase the speed of rotation of therotatable member. The robot 1000 can drive the rotatable member torotate at a third speed of rotation. The third speed of rotation can bethe same as or similar to the first speed of rotation. In someimplementations, the third speed of rotation is greater than the secondspeed of rotation and less than the first speed of rotation. The robot1000 can reactivate the drive system after the robot 1000 moves beyondthe edges of the rug 1006 or beyond the tassels 1012 into an interior ofthe rug 1006. The robot 1000 can be controlled to move over tassels inexamples in which tassels surround an interior of a rug 1006. Forexample, tassels can be positioned along an entire perimeter of the rug.As the robot 1000 moves off of the rug 1006, the robot 1000 can operatethe drive system of the rotatable members so that the rotatable membersrotate as the robot 1000 moves over the edges of the rug 1006 or thetassels 1012 of the rug 1006. This allows the robot 1000 to clean aregion along a perimeter of the rug 1006. The robot 1000 can drive therotatable members at a fourth speed of rotation. In someimplementations, the fourth speed of rotation is the same as the thirdspeed of rotation. In some implementations, the fourth speed of rotationis greater than the second speed of rotation.

Objects and obstacles represented in these images as described hereincan be overlaid on this three-dimensional representation, therebyaccurately depicting placement of doors and windows in the environment20. Alternatively or additionally, machine learning techniques can beused to detect distances of objects from the robot 700 that appear in asingle image captured by the robot 700. Based on these distances, athree-dimensional representation can be generated to be presented to theuser on the mobile device 652.

As described herein, objects and obstacles can be represented in imagescaptured by the robot 700 and can be overlaid on a three-dimensionalrepresentation. In some implementations, referring to FIG. 16A, a list1600 of objects, including debris, obstacles, or other objects,encountered by an autonomous cleaning robot (e.g., similar to the robot700) can be presented to a user through a mobile device 1602 (e.g.,similar to the mobile device 652 described herein). The list 1600 canidentify a name 1604 of an object, a location 1606 of the object, and atime 1608 that the object was encountered by the robot. Referring toFIG. 16B, the mobile device 1602 can, for example, present arepresentation of an object encountered by the robot. In the exampledepicted in FIG. 16B, the object is a cord 1610. The representation ofthe object can be generated based on the imagery captured by the robot.The representation of the object can be generated based on arepresentation of the object in imagery captured by the robot 700. Insome examples, the representation of the object presented on the mobiledevice 1602 can correspond to a portion of the imagery captured by therobot. Alternatively or additionally, the object can be identified fromthe imagery of the robot, and then a computer-generated image or a stockimage can be presented as part of the logged object presented to theuser through the mobile device 1602. For example, the robot canencounter different types of objects during a mission. The robot cancapture imagery of objects ingested by the robot or encountered by therobot during the mission. Portions of the captured imagery representingthe object can be presented by the mobile device 1602, e.g., the cord1610 shown in FIG. 16B, or the object can be identified using theimagery and then representations of the object can be presented on themobile device 1602.

A representation of the object can be presented on the mobile device1602, and the mobile device 1602 can issue a request for the user toconfirm the identity of the object. For example, if the object is thecord 1610, the mobile device 1602 can present the representation of thecord 1610 and ask the user to confirm that the object is a cord. In someimplementations, the mobile device 1602 can provide a list of types ofobjects detected and/or ingested by the robot 700, and in someimplementations, the mobile device 1602 can provide indicators, e.g.,overlaid on the stitched image representation of the floor surfacedescribed herein, of locations of the objects detected and/or ingestedby the robot. For example, as shown in FIG. 16B, in addition topresenting imagery 1612 representing the cord 1610, the mobile device1602 can present a top view representation 1614 of an environment of therobot, and can provide an indicator 1616 overlaid on the representation1614 to indicate the location where the cord 1610 was encountered. Themobile device 1602 can identify the object as the cord 1610 and requestthat the user confirm that the object is indeed the cord 1610. Byidentifying the object and presenting the indicator 1616 of the locationof the object, the mobile device 1602 can allow a user to easily tidy upa room so that the robot can avoid the object in a future mission.

The method 1100 depicted in FIG. 11 is described in connection with thecliff 1208 depicted in FIG. 12. In some implementations, the method 1100can be used for avoiding steep drop-offs in the environment 1202, e.g.,floor surfaces that form an angle greater than 45 degrees relative to ahorizontal plane.

In some implementations, rather than decreasing its speed as itapproaches a feature in the environment, an autonomous cleaning robotcan increase its speed in response to detecting a feature. For example,referring to FIG. 17, an autonomous cleaning robot 1700 can navigateabout a floor surface 1702 at a first speed while capturing imagery ofthe floor surface 1702. The robot 1700 can detect a raised portion 1704of the floor surface 1702, e.g., in a doorway or between differentrooms, based on the captured imagery. The robot 1200 can estimate adistance between it and the raised portion 1704 and increase its speedin response to being within a certain distance from the raised portion1704. For example, the robot 1700 moves at a first speed between a point1706 a and a point 1706 b, and then increases its speed at the point1706 b in response to determining that it is within the distance fromthe raised portion 1704. From the point 1706 b until the robot 1700reaches the threshold and moves over the raised portion 1704, the robot1700 travels at a second speed greater than the first speed. Thisincreased speed allows the robot 1700 to more easily travel over theraised portion 1704 without getting stuck on the raised portion 1704.

While an autonomous cleaning robot has been described herein, othermobile robots may be used in some implementations. For example, therobot 100 is a vacuum cleaning robot. In some implementations, anautonomous wet cleaning robot can be used. The robot can include a padattachable to a bottom of the robot, and can be used to perform cleaningmissions in which the robot scrubs the floor surface. The robot caninclude systems similar to those described with respect to the robot100. In some implementations, a patrol robot with an image capturedevice can be used. The patrol robot can include mechanisms to move theimage capture device relative to a body of the patrol robot. While therobot 100 is described as a circular robot, in other implementations,the robot 100 can be a robot including a front portion that issubstantially rectangular and a rear portion that is substantiallysemicircular. In some implementations, the robot 100 has an outerperimeter that is substantially rectangular.

Nevertheless, it will be understood that various modifications may bemade. Accordingly, other implementations are within the scope of theclaims.

What is claimed is:
 1. A method comprising: capturing, by an imagecapture device on an autonomous cleaning robot, imagery of a portion ofa floor surface forward of the autonomous cleaning robot, the portion ofthe floor surface comprising at least a portion of a rug; andmaneuvering the autonomous cleaning robot onto the rug along a pathselected based on the imagery of the portion of the floor surface. 2.The method of claim 1, wherein: the imagery of the portion of the floorsurface is indicative of a location of a tassel of the rug, andmaneuvering the autonomous cleaning robot onto the rug comprisesmaneuvering the autonomous cleaning robot onto the rug along the pathsuch that the autonomous cleaning robot avoids the tassel.
 3. The methodof claim 1, wherein: the path is a first path, the imagery of theportion of the floor surface is indicative of a direction along which atassel of the rug extends along the floor surface, and the methodfurther comprises maneuvering the autonomous cleaning robot off of therug along a second path such that the autonomous cleaning robot movesover the tassel in a direction substantially parallel to the directionalong which the tassel extends.
 4. The method of claim 1, wherein: theimagery of the portion of the floor surface is indicative of a locationof a corner of the rug, and maneuvering the autonomous cleaning robotonto the rug comprises maneuvering the autonomous cleaning robot ontothe rug along the path such that the autonomous cleaning robot avoidsthe corner of the rug.
 5. The method of claim 1, wherein: the imagerycomprises a plurality of images, and maneuvering the autonomous cleaningrobot onto the rug along a path selected based on the imagery of theportion of the floor surface comprises maneuvering the autonomouscleaning robot onto the rug along the path selected based on a locationof an edge of the rug represented in the plurality of images.
 6. Anautonomous cleaning robot comprising: a drive system to support theautonomous cleaning robot above a floor surface, the drive system beingoperable to maneuver the autonomous cleaning robot about the floorsurface; an image capture device positioned on the autonomous cleaningrobot to capture imagery of a portion of the floor surface forward ofthe autonomous cleaning robot; and a controller operably connected tothe drive system and the image capture device, wherein the controller isconfigured to execute instructions to perform operations comprising:maneuvering the autonomous cleaning robot at a first speed along a firstportion of the floor surface toward a second portion of the floorsurface, the second portion of the floor surface having a lowerelevation than the first portion of the floor surface, detecting thesecond portion of the floor surface based on the imagery captured by theimage capture device, and maneuvering the autonomous cleaning robot at asecond speed along the first portion of the floor surface toward thesecond portion of the floor surface after detecting the second portionof the floor surface, wherein the second speed is less than the firstspeed.
 7. The autonomous cleaning robot of claim 6, wherein maneuveringthe autonomous cleaning robot at the second speed along the firstportion of the floor surface after detecting the second portion of thefloor surface comprises: initiating reduction of a speed from theautonomous cleaning robot from the first speed to the second speed basedon determining, from the imagery captured by the image capture device,the autonomous cleaning robot is no more than a distance from the secondportion of the floor surface.
 8. The autonomous cleaning robot of claim7, wherein the distance is between 50% to 300% of a length of theautonomous cleaning robot.
 9. The autonomous cleaning robot of claim 6,wherein the imagery captured by the image capture device represents atleast a portion of the second portion of the floor surface.
 10. Theautonomous cleaning robot of claim 6, comprising a single image capturedevice corresponding to the image capture device.
 11. The autonomouscleaning robot of claim 6, wherein the image capture device is directedat an angle between 10 and 30 degrees above the floor surface.
 12. Theautonomous cleaning robot of claim 11, wherein a horizontal field ofview of the image capture device is between 90 and 150 degrees.
 13. Theautonomous cleaning robot of claim 6, further comprising a cliff sensordisposed on a bottom portion of the autonomous cleaning robot, the cliffsensor configured to detect the second portion of the floor surface asthe bottom portion of the autonomous cleaning robot moves over thesecond portion of the floor surface.
 14. The autonomous cleaning robotof claim 13, wherein the operations comprise maneuvering the autonomouscleaning robot along the first portion of the floor surface away fromthe second portion of the floor surface as the cliff sensor detects thesecond portion of the floor surface.
 15. An autonomous cleaning robotcomprising: a drive system to support the autonomous cleaning robotabove a floor surface, the drive system being operable to maneuver theautonomous cleaning robot about the floor surface; and a controlleroperably connected to the drive system, wherein the controller isconfigured to execute instructions to perform operations comprising:maneuvering the autonomous cleaning robot at a first speed along a firstportion of the floor surface toward a second portion of the floorsurface, the second portion of the floor surface having a lowerelevation than the first portion of the floor surface, and after theautonomous cleaning robot is within a distance from the second portionof the floor surface, maneuvering the autonomous cleaning robot at asecond speed along the first portion of the floor surface based on theautonomous mobile, wherein the second speed is less than the firstspeed.
 16. The autonomous cleaning robot of claim 15, wherein thedistance is between 50% to 300% of a length of the autonomous cleaningrobot.
 17. The autonomous cleaning robot of claim 15, whereinmaneuvering the autonomous cleaning robot at the second speed along thefirst portion of the floor surface comprises: initiating reduction of aspeed from the autonomous cleaning robot from the first speed to thesecond speed based on determining the autonomous cleaning robot iswithin the distance from the second portion of the floor surface. 18.The autonomous cleaning robot of claim 15, further comprising a cliffsensor disposed on a bottom portion of the autonomous cleaning robot,the cliff sensor configured to detect the second portion of the floorsurface as the bottom portion of the autonomous cleaning robot movesdirectly over the second portion of the floor surface.
 19. Theautonomous cleaning robot of claim 18, wherein the operations comprisemaneuvering the autonomous cleaning robot along the first portion of thefloor surface away from the second portion of the floor surface as thecliff sensor detects the second portion of the floor surface.
 20. Amethod comprising: maneuvering an autonomous cleaning robot at a firstspeed along a first portion of a floor surface toward a second portionof the floor surface, the second portion of the floor surface having alower elevation than the first portion of the floor surface; detecting,using an image capture device positioned on the autonomous cleaningrobot to capture imagery of a portion of the floor surface forward ofthe autonomous cleaning robot, the second portion of the floor surface;and maneuvering the autonomous cleaning robot at a second speed alongthe first portion of the floor surface toward the second portion of thefloor surface after detecting the second portion of the floor surface,wherein the second speed is less than the first speed.
 21. The method ofclaim 20, further comprising maneuvering the autonomous cleaning robotalong the first portion of the floor surface away from the secondportion of the floor surface as a cliff sensor of the autonomouscleaning robot detects the second portion of the floor surface.